1. Field of the Invention
The invention relates to the processing of data from signals that indicate information related to scatterers, such as signals from a Doppler scanning system, and in particular, to multi-stage processing that estimates the spectral moments of the scatterers enabling an effective trade-off between processing resources and accuracy.
2. Statement of the Problem
Doppler systems detect weather phenomena by reflecting signals off of rain, snow, dust, and debris carried by the wind. In a Doppler radar system, a transmitter emits a series of pulses that are scattered by these materials (scatterers). Some of the energy in the scattered pulses is reflected to a receiver. For a given pointing direction (azimuth and elevation), these pulses are sampled by the receiver as a function of time. Due to the propagation of the transmitted pulses, the time-sampling gives information as a function of range from the receiver. Often times, the transmitter is rotated, so that the continuous set of pulses also gives information as a function of the rotation angle. Typically, radars rotate around a vertical axes (conical, or azimuthal scanning) or along a plane perpendicular to the ground (scanning in elevation). Thus, Doppler systems obtain data that indicates the presence and motion of scatterers at various ranges and angles from the receiver. Doppler systems process this data to estimate the moments (i.e. the first three moments of the Doppler spectrum). The zeroth Doppler moment is related to the average intensity of the reflected energy of the scatterers. The first Doppler moment is related to the reflectivity-weighted velocity of the scatterers toward or away from the receiver (radial velocity). The second Doppler moment is related to the reflectivity-weighted variance of the relative motion of the scatterers. In this manner, Doppler systems are able to estimate wind patterns to aide in the detection of phenomena such as severe storms, tornadoes and turbulence.
Ideally, Doppler systems would only receive reflected energy that is associated with weather phenomenon, but unfortunately, Doppler systems also receive reflections from birds, insects, ground clutter, and other contaminants. To complicate the situation, Doppler systems are also affected by many types of spurious signals. Additional processing is required if atmospheric signals are to be effectively analyzed without corruption from contaminants.
Another problem faced by Doppler systems is the production of quality weather data in a timely manner. For phenomena such as severe storms and tornadoes, real time detection is essential. Doppler scanning at multiple azimuths and ranges generates a huge volume of data that can be contaminated by spurious reflections and signals. Typical real-time Doppler processing systems are simplistic and prone to errors in the presence of contamination. Existing Doppler processing systems that are sophisticated enough to eliminate these errors are too slow to handle real-time operation with vast quantities of data.
Another challenge faced by Doppler systems is the production of confidence values for the output data. The confidence values indicate a probability that the output data (e.g., the Doppler moments) is accurate. Systems that process the output data may use the confidence values to discount or ignore potentially inaccurate data. Currently, fast Doppler processing systems may weed-out some bad data based on signal-to-noise (SNR) thresholds, but these systems do not have the processing time or capacity to provide effective confidence values. Furthermore, this use of strict thresholding only provides for a binary (good/bad) indicator of the data quality. Currently, sophisticated Doppler systems that provide confidence values for their outputs are too slow to handle real-time operation with vast quantities of data.
Detailed Discussion of the Problemxe2x80x94FIGS. 1-5
A problem for the Doppler system is the generation of accurate moments in real-time. The large volume of data produced when scanning Doppler measurement devices are used in applications, such as turbulence sensing and warning, and weather analysis, precludes the use of the accurate real-time processing of moments by highly sophisticated moment estimation methods. The typical real-time methods for processing of scanning radar and lidar data for meteorological applications are fast, but inaccurate at low signal-to-noise ratio (SNR) or in the presence of clutter, radio frequency interference (RFI) and other contaminants.
Radars and lidars use a transmitter and receiver to measure the radial motion of scatterers within the radar/lidar pulse volumes. The transmitter sends out a series of pulses in a given direction. A Fourier transform is applied to the complex time series of received amplitudes and phases for a given range to produce a Doppler spectrum for that range (see FIG. 1). The spectra give information regarding the return power as a function of Doppler frequency. These frequencies are directly related to the Doppler radial velocities. As the device scans, a sequence of Doppler spectra is generated. The term xe2x80x9crange gatexe2x80x9d will be used to describe the range and pointing direction. In the following and without any loss of generality, it will be assumed that the radar is scanning in azimuth. FIG. 2 shows a contour plot of Doppler spectra before processing for each range gate in a single pointing direction (azimuth).
Let the portion of the spectrum that is primarily influenced by the scatterers of interest (e.g., return from atmospheric scatterers), be referred to as the xe2x80x9csignal.xe2x80x9d If all of the scatterers were identical and moved at the same radial velocity, then the spectra would have significant amplitude only at the frequency that corresponds to that radial velocity. The amplitude of the signal would be proportional to the size and number of scatterers in the pulse interaction volume, the shape of this volume, and the transmitted power from the Doppler measurement device. If the scatterers have more than one velocity, the signal broadens. Theoretically, this broadened signal takes the form of a Gaussian shaped spectra, as is shown in FIG. 3. The width of the signal is proportional to the variance in radial velocities in the pulse volume, which in turn, is indicative of turbulence. The width of the signal can be measured using either the second moment of the Doppler spectrum or the width of a Gaussian fit to the signal. The first moment or the center of a Gaussian fit can be used to measure the mean velocity of the scatterers. The signal power is proportional to the size and distribution of the scatterers in the pulse volume, and is given by the area of the Gaussian, or the zeroth moment.
For actual signals, the velocity mean and width are not always easy to calculate. The Doppler measurement systems have noise associated with the measurements, which contaminates the spectrum and may cause the signal to deviate from the theoretical Gaussian shape. Systematic variations in particle size and/or constitution (e.g. rain or ice) over the pulse volume can also cause the signal to lose its ideal shape. For weaker signals, the noise level can make the signal very difficult to identify. Techniques (e.g., phase randomization) to minimize the effect of scattering returns from beyond the device""s maximum unambiguous range (second trip returns) can elevate the noise level and introduce random fluctuations onto the signal. Clutter, RFI, non-atmospheric objects, such as birds and insects, and radar anomalies can produce contamination in the spectra for radar. Other Doppler measurement devices, (e.g. acoustic radars, or sodars), have similar sources of contamination. The task is then to identify the atmospheric signal in the spectra and minimize the effect of contamination on the moment calculations.
Current Moment Estimation Methods
Current moment estimation methods fall into two categories: fast or highly accurate. The so-called pulse-pair and peak-picking methods are fast, but they do not address the contamination problems. The highly accurate National Center for Atmospheric Research (NCAR) Improved Moments Algorithm (NIMA), typically used for Doppler wind profilers, does a very good job of addressing contamination problems and utilizing quality control, however it is too computationally inefficient for use in most existing scanning Doppler measurement systems.
Pulse-Pair Method
The fastest method for determining spectral moments is the so-called pulse-pair method. The pulse-pair method is described in Doppler Radar and Weather Observations, by R. J. Doviak and D. S. Zrnic, Academic Press, 1993. The time-domain pulse-pair method computes the moments directly from the complex time-domain signal, without requiring spectra to be computed. For moderate to high SNR and little contamination from other sources, first moment estimates using pulse-pair methods are usually accurate. On the other hand, second moment estimates are much more sensitive to SNR and are also easily corrupted by signal contamination. These problems in moment estimation under low SNR conditions can be seen in the moment estimations by the pulse-pair method for the upper range gates of FIG. 4.
On FIG. 4, note that each range gate (horizontal slice) represents the amplitude of a spectrum as a function of Doppler velocity. The cross marks the first moment. The bar is indicative of the second moment. The more distant range gates have lower SNR and hence, inaccurate moment estimations.
The pulse-pair method finds the first moment of the spectrum by assuming that there is only one signal in the spectrum and using the argument of the complex autocorrelation function. Assuming that the autocorrelation has a Gaussian shape, the second moment can be determined from the logarithm of the ratio of the signal power to the magnitude of the autocorrelation at one lag. The moments will be in error if the SNR is low, if contamination signals are present, or if the signal is non-Gaussian.
Quality control can be done with the SNR, (i.e., throwing out moments when the SNR is below a specified value). The low SNR case is easy to detect, but the influence of multiple signals, or a single signal that does not resemble a Gaussian, is much harder to detect in the time domain. If the data is transformed to the spectral domain, much of the speed advantage is lost, but quality control algorithms that can consider the shape of the signal become possible. In the spectral domain, the peak-picking method is generally more accurate than the pulse pair method.
Peak-Picking Method
In the spectral domain, the most effective of the fast methods is the so-called peak-picking method. Assuming that only one signal is present in the spectrum, the peak amplitude in the spectrum is the peak of the signal, as can be seen in FIG. 3. The first and second moments are calculated using the spectral region around the peak with either integration or a fit to a Gaussian. The signal region is usually determined by the bounds at the velocity bins (so-called velocity xe2x80x9ccut-offsxe2x80x9d) where the spectrum crosses the noise level nearest to the peak in both increasing and decreasing velocity directions, as seen in FIG. 3.
FIG. 5 shows moments estimated by the peak-picking method that are more accurate than those estimated by the pulse-pair method (FIG. 4) for low SNR signals. On FIG. 5, note that each range gate represents the amplitude of a spectrum. The cross marks the first moment. The bar is indicative of the second moment.
The peak-picking method and the pulse-pair method both have problems when contaminants create spurious peaks in the spectra. NIMA, in essence is a peak-picking method, however, it uses sophisticated processing to find the desired signal region and the proper velocity cut-offs, even in the presence of significant contamination. NIMA also uses more elaborate quality control than the SNR quality control often used with the fast moment estimation methods.
Quality Control Using SNR
In addition to the problems involved in estimating the moments of the spectra, there is also the problem of determining the reliability of the estimates. If the scanning systems are to be used for evaluation of turbulence or other hazards, then estimates from questionable data need to be removed or discounted. Current methods such as the pulse-pair and peak-picking methods can use strict SNR thresholds to remove poor quality estimates, but that measure does not address poor quality estimates from multiple signals in spectra with above-threshold SNR. NIMA does more sophisticated quality control, calculating confidence values using SNR as well as many other indicators. NIMA was originally developed for profiler systems, which due to a relatively low data rate, permit more sophisticated processing than scanning Doppler measurement systems.
Processing for Profilers Versus Scanning Doppler Measurement Systems
Doppler spectra from scanning Doppler measurement systems are more difficult to extract accurate moments from than spectra from profilers. Firstly, the time constraints are much more stringent. Profilers dwell along one pointing direction for time periods of many seconds to a few minutes, whereas scanning systems frequently process an azimuth in time periods of tens of milliseconds. The second large difference is that fewer spectra are available for averaging and/or filtering in scanning systems. Profilers, which compute many individual spectra in the dwell time, average over many spectra. Scanning systems are restricted in the number of spectra that could be averaged by the motion of the Doppler measurement device and/or the lack of overlap between some adjacent pulse volumes. Second-trip returns are also far less of a problem for profilers. The profiler pulses are less likely to encounter multiple cloud structures when they are traveling vertically, than the scanning radar and lidar pulses that travel horizontally or obliquely. Some problems that profilers encounter are less prominent in scanning systems, however. For example, RFI is much less frequent in scanning radars and lidars than in profilers, but contamination can still be a problem, in general.
Spectral Smoothing
Due to the random motion of the scatterers, a single, raw Doppler spectrum is xe2x80x9cnoisy,xe2x80x9d resulting in problems with accurate moment estimation. This xe2x80x9cnoisexe2x80x9d is not only from systematic hardware noise, but is also due to the random motion of the scatterers relative to each other. Averaging spectra over space and/or time can alleviate a certain amount of this problem. Due to natural inhomogeneities and non-stationarities in the signal field, only a limited number of spectra should be averaged. Hence, the effect that averaging over a small number of samples has on reducing the noise from the scatterers is limited. None-the-less, spectral averaging helps in the detectability of the signal. Depending on the processing resources, a median filter may be used to great benefit over simple averaging. This is because the median filter is more robust than averaging when the data contains outliers.
The averaging of spectra with scanning Doppler measurement devices is not a standard practice. If there is rapid motion by the scanning platform, temporal averaging cannot be performed. For a stationary or slowly moving platform with return times commensurate with approximate stationarity in the data, averaging can be performed over time, if the range gates are offset to account for the differences in locations of the scatterers relative to the range gates. This offsetting is difficult because the differences in locations will not be perfectly known or necessarily be an integral number of range gates. If approximate homogeneity of the scatterers holds between neighboring range gates, averaging along range can be performed. For ranges relatively close to the device where approximate homogeneity holds, averaging across azimuths can also be performed.
NIMA
NIMA overcomes most of the accuracy problems of the pulse-pair method and the peak-picking method by employing fuzzy logic and image processing techniques to recognize the two-dimensional (Doppler velocity and range) regions of the spectra (cf. FIG. 4) that contain information about the desired signal. Unfortunately, NIMA is too computationally complex to run in real-time for scanning Doppler measurement devices.
NIMA approaches moment estimation as primarily a pattern recognition problem. That is, NIMA identifies the region of the spectra that corresponds to the atmospheric signal using fuzzy logic image processing methods. The atmospheric signal is usually clearly identified as a region that extends through many range gates. The RFI and clutter signals can also create regions that extend through range, but these regions are often much narrower than the atmospheric signal. NIMA uses fuzzy logic image processing methods to differentiate between these contaminant features and the desired atmospheric ones. NIMA uses a series of tests on the data to determine the extent of the atmospheric signal region and eliminate noise or contaminants. This process can be computationally intensive, but results in very accurate moment estimates, even for highly contaminated signals and for some low SNR signals.
In addition to producing more accurate moment estimates by identifying the signal region as distinct from the contaminants, NIMA also generates sophisticated quality control or confidence values. A very sophisticated process is used in NIMA to obtain the confidences, which is generally too computationally expensive for application to scanning Doppler measurement devices.
The problem at hand is to develop a method of combining the fast signal processing techniques with the some of more sophisticated quality techniques to produce the most accurate moment estimates possible given limited processing capabilities.
The invention helps solve the problem of combining the fast methods with sophisticated quality control by implementing a multistage computation method. The multistage computation method calculates moments from the parts of the spectra that contain information about the signal, while disregarding the portions of the spectra that are shaped primarily by contaminants. The use of quality measures allows for efficient use of limited processing time on computations that are most likely to improve the accuracy of the moment estimates. For Doppler measurement devices, this multistage computation method is more flexible and generally more accurate than the other fast moment calculation methods, and is faster than the image processing method used with profilers. The multistage processing method consistently produces better moments than the pulse-pair method or simple peak-picking, especially for low SNR signals and clearly identifiable contaminants. The multistage processing method also produces confidences that range in value from zero to one for the moment estimates, which are more meaningful than binary results of SNR thresholding.
Examples of the invention include methods of operation, moment estimation systems, and software products. In these examples, a receiver receives signals that indicate information related to scatterers. One example of these signals comprises reflected energy from a Doppler scanning system. The moment estimation systems include a processing system and an interface that receives data from the signals. The software product includes software that directs a processing system, and a storage system that stores the software.
In some examples of the invention, the processing system receives data from the signals and tracks allowed processing time for the data. The processing system performs a first stage of processing for the data to generate first estimates of spectral moments for the signals. The processing system performs additional stages of processing for the data as the allowed processing time permits and stops the additional stages of processing for the data when the allowed processing time expires. The additional stages of processing may comprise generating second estimates for at least some of the spectral moments.
In some examples of the invention, the processing system receives data from the signals. The processing system tracks allowed processing time and processes the data to generate first estimates of spectral moments for the signals. The processing system generates first confidence factors representing quality measures for the first estimates of the spectral moments. If the allowed processing time permits, the processing system selects selected ones of the spectral moments based on the first confidence factors and generates second estimates of the selected ones of the spectral moments.
The processing system may average or median filter the data to reduce noise. The processing system may calculate noise levels and identify spectral points for use in moment estimation. The processing system may generate the first confidence factors based on consistency across at least one of: range, pointing direction, and time.
The processing system may select the selected ones of the spectral moments that have lower first confidence factors relative to neighboring ones of the spectral moments. The processing system may generate estimated noise levels and cut-offs for the selected ones of the spectral moments based on calculated noise levels and cut-offs for the neighboring ones of the spectral moments. The processing system may use the estimated noise levels and cut-offs to identify spectral points for use in moment re-estimation.
The processing system may generate second confidence factors representing quality measures for the second estimates of the selected ones the spectral moments if the allowed processing time permits. The processing system may generate the second confidence factors based on consistency across at least one of: range, pointing direction, and time. The processing system may generate third estimates of some of the selected ones of the spectral moments based on the second confidence factors if the allowed processing time permits.